已经证明，经过代码完成培训的大型语言模型（LLMS）能够合成DocStrings的简单Python程序[1]。我们发现这些代码编写的LLM可以被重新使用以编写机器人策略代码，给定自然语言命令。具体而言，策略代码可以表达处理感知输出的功能或反馈循环（例如，从对象检测器[2]，[3]）并参数化控制原始API。当作为输入提供了几个示例命令（格式为注释）后，然后是相应的策略代码（通过少量提示），LLMS可以接收新命令并自主重新编写API调用以分别生成新的策略代码。通过链接经典的逻辑结构并引用第三方库（例如，numpy，shapely）执行算术，以这种方式使用的LLM可以编写（i）（i）表现出空间几何推理的机器人策略，（ii）（ii）将其推广到新的说明和新指令和新指令和（iii）根据上下文（即行为常识）规定模棱两可的描述（例如“更快”）的精确值（例如，速度）。本文将代码作为策略介绍：语言模型生成程序的以机器人为中心的形式化（LMP），该程序可以代表反应性策略（例如阻抗控制器），以及基于Waypoint的策略（基于远见的选择，基于轨迹，基于轨迹，控制），在多个真实的机器人平台上展示。我们方法的核心是促使层次代码 - 代码（递归定义未定义的功能），该代码可以编写更复杂的代码，还可以改善最新的代码，以解决HOMANEVAL [1]基准中的39.8％的问题。代码和视频可从https://code-as-policies.github.io获得。

最近的作品表明，如何将大语言模型（LLM）的推理能力应用于自然语言处理以外的领域，例如机器人的计划和互动。这些具体的问题要求代理商了解世界上许多语义方面：可用技能的曲目，这些技能如何影响世界以及对世界的变化如何映射回该语言。在体现环境中规划的LLMS不仅需要考虑要做什么技能，还需要考虑如何以及何时进行操作 - 答案随着时间的推移而变化，以响应代理商自己的选择。在这项工作中，我们调查了在这种体现的环境中使用的LLM在多大程度上可以推论通过自然语言提供的反馈来源，而无需任何其他培训。我们建议，通过利用环境反馈，LLM能够形成内部独白，使他们能够在机器人控制方案中进行更丰富的处理和计划。我们研究了各种反馈来源，例如成功检测，场景描述和人类互动。我们发现，闭环语言反馈显着改善了三个领域的高级指导完成，包括模拟和真实的桌面顶部重新排列任务以及现实世界中厨房环境中的长途移动操作任务。

机器人需要在约束环境（例如架子和橱柜）中操纵物体，以帮助人类在房屋和办公室等日常设置中。这些限制因减少掌握能力而变得难以操纵，因此机器人需要使用非忽视策略来利用对象环境联系来执行操纵任务。为了应对在这种情况下规划和控制接触性富裕行为的挑战，该工作使用混合力量速度控制器（HFVC）作为技能表示和计划的技能序列，并使用学到的先决条件进行了计划。尽管HFVC自然能够实现稳健且合规的富裕行为，但合成它们的求解器传统上依赖于精确的对象模型和对物体姿势的闭环反馈，这些反馈因遮挡而在约束环境中很难获得。我们首先使用HFVC综合框架放松了HFVC对精确模型和反馈的需求，然后学习一个基于点云的前提函数，以对HFVC执行仍将成功地进行分类，尽管建模不正确。最后，我们在基于搜索的任务计划者中使用学到的前提来完成货架域中的接触式操纵任务。我们的方法达到了$ 73.2 \％$的任务成功率，表现优于基线实现的$ 51.5 \％$，而没有学习的先决条件。在模拟中训练了前提函数时，它也可以转移到现实世界中，而无需进行其他微调。

作为自治机器人的互动和导航在诸如房屋之类的真实环境中，可靠地识别和操纵铰接物体，例如门和橱柜是有用的。在对象铰接识别中许多先前的作品需要通过机器人或人类操纵物体。虽然最近的作品已经解决了从视觉观测的预测，但他们经常假设根据其运动约束的铰接部件移动的类别级运动模型或观察序列的先验知识。在这项工作中，我们提出了Formnet，是一种神经网络，该神经网络识别来自RGB-D图像和分段掩模的单帧对象部分的对象部分之间的铰接机制。从6个类别的149个铰接对象的100K合成图像培训网络培训。通过具有域随机化的光保护模拟器呈现合成图像。我们所提出的模型预测物体部件的运动残余流动，并且这些流量用于确定铰接类型和参数。该网络在训练有素的类别中的新对象实例上实现了82.5％的铰接式分类精度。实验还展示了该方法如何实现新颖类别的泛化，并且在没有微调的情况下应用于现实世界图像。

This paper focuses on designing efficient models with low parameters and FLOPs for dense predictions. Even though CNN-based lightweight methods have achieved stunning results after years of research, trading-off model accuracy and constrained resources still need further improvements. This work rethinks the essential unity of efficient Inverted Residual Block in MobileNetv2 and effective Transformer in ViT, inductively abstracting a general concept of Meta-Mobile Block, and we argue that the specific instantiation is very important to model performance though sharing the same framework. Motivated by this phenomenon, we deduce a simple yet efficient modern \textbf{I}nverted \textbf{R}esidual \textbf{M}obile \textbf{B}lock (iRMB) for mobile applications, which absorbs CNN-like efficiency to model short-distance dependency and Transformer-like dynamic modeling capability to learn long-distance interactions. Furthermore, we design a ResNet-like 4-phase \textbf{E}fficient \textbf{MO}del (EMO) based only on a series of iRMBs for dense applications. Massive experiments on ImageNet-1K, COCO2017, and ADE20K benchmarks demonstrate the superiority of our EMO over state-of-the-art methods, \eg, our EMO-1M/2M/5M achieve 71.5, 75.1, and 78.4 Top-1 that surpass \textbf{SoTA} CNN-/Transformer-based models, while trading-off the model accuracy and efficiency well.

Decompilation aims to transform a low-level program language (LPL) (eg., binary file) into its functionally-equivalent high-level program language (HPL) (e.g., C/C++). It is a core technology in software security, especially in vulnerability discovery and malware analysis. In recent years, with the successful application of neural machine translation (NMT) models in natural language processing (NLP), researchers have tried to build neural decompilers by borrowing the idea of NMT. They formulate the decompilation process as a translation problem between LPL and HPL, aiming to reduce the human cost required to develop decompilation tools and improve their generalizability. However, state-of-the-art learning-based decompilers do not cope well with compiler-optimized binaries. Since real-world binaries are mostly compiler-optimized, decompilers that do not consider optimized binaries have limited practical significance. In this paper, we propose a novel learning-based approach named NeurDP, that targets compiler-optimized binaries. NeurDP uses a graph neural network (GNN) model to convert LPL to an intermediate representation (IR), which bridges the gap between source code and optimized binary. We also design an Optimized Translation Unit (OTU) to split functions into smaller code fragments for better translation performance. Evaluation results on datasets containing various types of statements show that NeurDP can decompile optimized binaries with 45.21% higher accuracy than state-of-the-art neural decompilation frameworks.

Image Virtual try-on aims at replacing the cloth on a personal image with a garment image (in-shop clothes), which has attracted increasing attention from the multimedia and computer vision communities. Prior methods successfully preserve the character of clothing images, however, occlusion remains a pernicious effect for realistic virtual try-on. In this work, we first present a comprehensive analysis of the occlusions and categorize them into two aspects: i) Inherent-Occlusion: the ghost of the former cloth still exists in the try-on image; ii) Acquired-Occlusion: the target cloth warps to the unreasonable body part. Based on the in-depth analysis, we find that the occlusions can be simulated by a novel semantically-guided mixup module, which can generate semantic-specific occluded images that work together with the try-on images to facilitate training a de-occlusion try-on (DOC-VTON) framework. Specifically, DOC-VTON first conducts a sharpened semantic parsing on the try-on person. Aided by semantics guidance and pose prior, various complexities of texture are selectively blending with human parts in a copy-and-paste manner. Then, the Generative Module (GM) is utilized to take charge of synthesizing the final try-on image and learning to de-occlusion jointly. In comparison to the state-of-the-art methods, DOC-VTON achieves better perceptual quality by reducing occlusion effects.

In recent years, the Transformer architecture has shown its superiority in the video-based person re-identification task. Inspired by video representation learning, these methods mainly focus on designing modules to extract informative spatial and temporal features. However, they are still limited in extracting local attributes and global identity information, which are critical for the person re-identification task. In this paper, we propose a novel Multi-Stage Spatial-Temporal Aggregation Transformer (MSTAT) with two novel designed proxy embedding modules to address the above issue. Specifically, MSTAT consists of three stages to encode the attribute-associated, the identity-associated, and the attribute-identity-associated information from the video clips, respectively, achieving the holistic perception of the input person. We combine the outputs of all the stages for the final identification. In practice, to save the computational cost, the Spatial-Temporal Aggregation (STA) modules are first adopted in each stage to conduct the self-attention operations along the spatial and temporal dimensions separately. We further introduce the Attribute-Aware and Identity-Aware Proxy embedding modules (AAP and IAP) to extract the informative and discriminative feature representations at different stages. All of them are realized by employing newly designed self-attention operations with specific meanings. Moreover, temporal patch shuffling is also introduced to further improve the robustness of the model. Extensive experimental results demonstrate the effectiveness of the proposed modules in extracting the informative and discriminative information from the videos, and illustrate the MSTAT can achieve state-of-the-art accuracies on various standard benchmarks.

It has been observed in practice that applying pruning-at-initialization methods to neural networks and training the sparsified networks can not only retain the testing performance of the original dense models, but also sometimes even slightly boost the generalization performance. Theoretical understanding for such experimental observations are yet to be developed. This work makes the first attempt to study how different pruning fractions affect the model's gradient descent dynamics and generalization. Specifically, this work considers a classification task for overparameterized two-layer neural networks, where the network is randomly pruned according to different rates at the initialization. It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero and the network exhibits good generalization performance. More surprisingly, the generalization bound gets better as the pruning fraction gets larger. To complement this positive result, this work further shows a negative result: there exists a large pruning fraction such that while gradient descent is still able to drive the training loss toward zero (by memorizing noise), the generalization performance is no better than random guessing. This further suggests that pruning can change the feature learning process, which leads to the performance drop of the pruned neural network. Up to our knowledge, this is the \textbf{first} generalization result for pruned neural networks, suggesting that pruning can improve the neural network's generalization.

This work studies training one-hidden-layer overparameterized ReLU networks via gradient descent in the neural tangent kernel (NTK) regime, where, differently from the previous works, the networks' biases are trainable and are initialized to some constant rather than zero. The first set of results of this work characterize the convergence of the network's gradient descent dynamics. Surprisingly, it is shown that the network after sparsification can achieve as fast convergence as the original network. The contribution over previous work is that not only the bias is allowed to be updated by gradient descent under our setting but also a finer analysis is given such that the required width to ensure the network's closeness to its NTK is improved. Secondly, the networks' generalization bound after training is provided. A width-sparsity dependence is presented which yields sparsity-dependent localized Rademacher complexity and a generalization bound matching previous analysis (up to logarithmic factors). As a by-product, if the bias initialization is chosen to be zero, the width requirement improves the previous bound for the shallow networks' generalization. Lastly, since the generalization bound has dependence on the smallest eigenvalue of the limiting NTK and the bounds from previous works yield vacuous generalization, this work further studies the least eigenvalue of the limiting NTK. Surprisingly, while it is not shown that trainable biases are necessary, trainable bias helps to identify a nice data-dependent region where a much finer analysis of the NTK's smallest eigenvalue can be conducted, which leads to a much sharper lower bound than the previously known worst-case bound and, consequently, a non-vacuous generalization bound.